Computing device and extended reality integration

ABSTRACT

Techniques are provided for integrating mobile device and extended reality experiences. Extended reality technologies can include virtual reality (VR), augmented reality (AR), mixed reality (MR), etc. In some examples, a synthetic (or virtual) representation of a device (e.g., a mobile device, such as a mobile phone or other type of device) can be generated and displayed along with VR content being displayed by a VR device (e.g., a head-mounted display (HMD)). In another example, content from the device (e.g., visual content being displayed and/or audio content being played by the device) can be output along with VR content being displayed by the VR device. In another example, one or more images captured by a camera of the device and/or audio obtained by a microphone of the device can be obtained from the device by a virtual reality device and can be output by the virtual reality device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.62/882,488, filed Aug. 3, 2019, which is hereby incorporated byreference, in its entirety and for all purposes.

TECHNICAL FIELD

The present disclosure generally relates to techniques and systems forproviding an integration of computing device and extended realityexperiences.

BACKGROUND

Extended reality technologies can be used to present virtual content tousers, and/or can combine real environments from the physical world andvirtual environments to provide users with extended reality experiences.The term extended reality can encompass virtual reality, augmentedreality, mixed reality, and the like. Each of these forms of extendedreality allows users to experience or interact with immersive virtualenvironments or content. For example, an extended reality experience canallow a user to interact with a real or physical environment enhanced oraugmented with virtual content. Extended reality technologies can beimplemented to enhance user experiences in a wide range of contexts,such as entertainment, healthcare, retail, education, social media, andso forth.

SUMMARY

In some examples, systems, methods, and computer-readable media aredescribed for integrating computing device and extended realityexperiences. For instance, the techniques described herein can allow auser enjoying an extended reality experience to seamlessly use acomputing device (e.g., a mobile device, such as a mobile phone, awearable device, or other type of device) or one or more functionalitiesof the computing device without interrupting the extended realityexperience.

In one illustrative example, a synthetic (or virtual) representation ofa user's computing device (e.g., a mobile device, such as a mobilephone, a wearable device, or other type of device) can be generated anddisplayed along with virtual reality content being displayed by avirtual reality device (e.g., a head-mounted display (HMD). In anotherexample, content from the computing device (e.g., content beingdisplayed by the device, a message received by the computing device, anotification generated by the computing device, and/or other content)can be displayed along with virtual reality content being displayed bythe virtual reality device. In another example, audio content from thecomputing device (e.g., a voice message, a video message, browsercontent, a phone call, streamed or locally-stored music, and/or otheraudio content) can be played by the virtual reality device (e.g.,through one or more speakers of the virtual reality device). In somecases, the audio content can be played by the virtual reality devicewhile displaying visual content from the computing device. In suchexamples, a user can view the computing device and/or the content fromthe computing device along with the virtual reality content, allowingthe user to interact with the computing device (e.g., control one ormore applications of a mobile phone) without removing the virtualreality device (e.g., while still wearing the HMD). The virtual realitycontent can continue to be displayed by the virtual reality device asthe synthetic representation of the computing device is displayed, orcan be paused while the synthetic representation of the computing deviceis displayed by the virtual reality device.

In another illustrative example, one or more images captured by a cameraof a computing device (e.g., a mobile device, such as a mobile phone, awearable device, or other type of device) can be obtained from thedevice by a virtual reality device. The virtual reality device candisplay the one or more images along with virtual reality content beingviewed through the virtual reality device (e.g., an HMD). In such anexample, the user can view the real world (as captured by the one ormore images) without removing the virtual reality device (e.g., whilestill wearing the HMD). In some cases, audio from the microphone of thecomputing device can be provided to the virtual reality device. Forexample, providing audio from the microphone of the computing device canallow the user to perceive what is going on in the real world withouthaving to remove the virtual reality device.

According to one illustrative example, a method of determining one ormore sizes of one or more objects is provided. The method includes:displaying, by a first device, virtual reality content; obtaining, bythe first device, a synthetic representation of a second device;displaying, by the first device, the synthetic representation of thesecond device with the virtual reality content; receiving, by the firstdevice, input requesting a change in functionality of the second device;and displaying, based on the input received by the first device, achange in the synthetic representation of the second device, the changein the synthetic representation of the second device representing thechange in functionality of the second device.

In another example, an apparatus for determining one or more sizes ofone or more objects is provided that includes a memory configured tostore one or more images and one or more processors implemented incircuitry and coupled to the memory. The one or more processors areconfigured to and can: cause virtual reality content to be displayed ona display; obtain a synthetic representation of a device; cause thesynthetic representation of the device to be displayed with the virtualreality content on the display; receive input requesting a change infunctionality of the device; and based on the received input, cause achange in the synthetic representation of the device to be displayed onthe display, the change in the synthetic representation of the devicerepresenting the change in functionality of the device

In another example, a non-transitory computer-readable medium isprovided that has stored thereon instructions that, when executed by oneor more processors, cause the one or more processor to: cause virtualreality content to be displayed by the first device; obtaining, by thefirst device, a synthetic representation of a second device; cause thesynthetic representation of the second device to be displayed with thevirtual reality content; receiving, by the first device, inputrequesting a change in functionality of the second device; and based onthe input received by the first device, cause a change in the syntheticrepresentation of the second device to be displayed by the first device,the change in the synthetic representation of the second devicerepresenting the change in functionality of the second device.

In another example, an apparatus for determining one or more sizes ofone or more objects is provided. The apparatus includes: means fordisplaying, by a first device, virtual reality content; means forobtaining, by the first device, a synthetic representation of a seconddevice; means for displaying, by the first device, the syntheticrepresentation of the second device with the virtual reality content;means for receiving, by the first device, input requesting a change infunctionality of the second device; and means for displaying, based onthe input received by the first device, a change in the syntheticrepresentation of the second device, the change in the syntheticrepresentation of the second device representing the change infunctionality of the second device.

In some aspects, the synthetic representation of the second deviceincludes a synthetic representation of a display of the second device,wherein content displayed by the second device is displayed in thesynthetic representation of the display of the second device.

In some aspects, the method, apparatuses, and computer-readable mediumdescribed above further comprise: receiving, by the first device, anindication of one or more inputs processed by the second device; anddisplaying, based on the indication of the one or more inputs processedby the second device, an additional change in the syntheticrepresentation of the second device, the additional change in thesynthetic representation of the second device representing an additionalchange in functionality of the second device.

In some aspects, the synthetic representation of the second device isoverlaid over the virtual reality content. In some aspects, the firstdevice includes a virtual reality head mounted display. In some aspects,the second device includes a mobile device.

In some aspects, the method, apparatuses, and computer-readable mediumdescribed above further comprise: obtaining, by the first device, audiocontent from the second device; and playing, by the first device, theaudio content from the second device.

In some aspects, the method, apparatuses, and computer-readable mediumdescribed above further comprise: obtaining, by the first device, audiocontent captured by a microphone of the second device; and playing, bythe first device, the audio content.

In some aspects, the method, apparatuses, and computer-readable mediumdescribed above further comprise: obtaining, by the first device, one ormore images captured by a camera of the second device; and displaying,by the first device, the one or more images with the virtual realitycontent. In some aspects, the one or more images are part of a videocaptured by the camera of the second device. In some aspects, the one ormore images are displayed with the virtual reality content as part ofthe synthetic representation of the second device. In some aspects, theone or more images are displayed within a synthetic representation of adisplay of the second device.

In some aspects, the method, apparatuses, and computer-readable mediumdescribed above further comprise: obtaining, by the first device, audiocontent captured by a microphone of the second device; and playing, bythe first device, the audio content while displaying the one or moreimages with the virtual reality content.

In some aspects, the method, apparatuses, and computer-readable mediumdescribed above further comprise: receiving a trigger; and displaying,based on receiving the trigger, the synthetic representation of thesecond device with the virtual reality content. In some aspects, thetrigger is based on information received from the second device.

In some aspects, the method, apparatuses, and computer-readable mediumdescribed above further comprise: receiving a removal trigger; andremoving, by the first device based on receiving the trigger, thesynthetic representation of the second device from display.

This summary is not intended to identify key or essential features ofthe claimed subject matter, nor is it intended to be used in isolationto determine the scope of the claimed subject matter. The subject mattershould be understood by reference to appropriate portions of the entirespecification of this patent, any or all drawings, and each claim.

The foregoing, together with other features and embodiments, will becomemore apparent upon referring to the following specification, claims, andaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Illustrative embodiments of the present application are described indetail below with reference to the following figures:

FIG. 1 is a diagram illustrating an example of virtual content displayedby a virtual reality device, in accordance with some examples;

FIG. 2 is a diagram illustrating an example of a syntheticrepresentation of a device displayed with virtual content, where thesynthetic representation of the device and the virtual content aredisplayed by a virtual reality device, in accordance with some examples;

FIG. 3 is a diagram illustrating an example of content of a devicedisplayed with virtual content, where the content of the device and thevirtual content are displayed by a virtual reality device, in accordancewith some examples;

FIG. 4 is a diagram illustrating an example of an image captured by adevice displayed with virtual content, where the image and the virtualcontent are displayed by a virtual reality device, in accordance withsome examples;

FIG. 5 is a diagram illustrating an example of switching between displayof virtual content and display of a real world by a virtual realitydevice, in accordance with some examples;

FIG. 6 is a diagram illustrating an example of components of a virtualreality device, in accordance with some examples;

FIG. 7 is a diagram illustrating an example of different display layers,in accordance with some examples;

FIG. 8 is a diagram illustrating another example of components of avirtual reality device, in accordance with some examples;

FIG. 9 is a flow diagram illustrating an example of a process fordisplaying virtual content, in accordance with some examples;

FIG. 10 is a flow diagram illustrating another example of a process fordisplaying virtual content, in accordance with some examples;

FIG. 11 is a flow diagram illustrating another example of a process fordisplaying virtual content, in accordance with some examples; and

FIG. 12 is an example computing device architecture of an examplecomputing device that can implement the various techniques describedherein.

DETAILED DESCRIPTION

Certain aspects and embodiments of this disclosure are provided below.Some of these aspects and embodiments may be applied independently andsome of them may be applied in combination as would be apparent to thoseof skill in the art. In the following description, for the purposes ofexplanation, specific details are set forth in order to provide athorough understanding of embodiments of the application. However, itwill be apparent that various embodiments may be practiced without thesespecific details. The figures and description are not intended to berestrictive.

The ensuing description provides exemplary embodiments only, and is notintended to limit the scope, applicability, or configuration of thedisclosure. Rather, the ensuing description of the exemplary embodimentswill provide those skilled in the art with an enabling description forimplementing an exemplary embodiment. It should be understood thatvarious changes may be made in the function and arrangement of elementswithout departing from the spirit and scope of the application as setforth in the appended claims.

As previously described, extended reality (XR) technologies can providevirtual content to a user, and/or combine real or physical environmentsand virtual environments (made up of virtual content) to provide userswith extended reality experiences. Extended reality experiences includevirtual reality (VR), augmented reality (AR), mixed reality (MR), and/orother immersive content. Examples described herein will use VR forillustrative purposes. However, one of ordinary skill will appreciatethat the techniques described herein can be applied to other extendedreality experiences, such as AR and MR. For example, in some cases, thetechniques described herein can be used for AR devices and/or MRdevices. An MR device can use video pass-through (e.g., as shown in anddescribed below with respect to FIG. 5) to render physical reality alongwith virtual content. However, due to the poor quality of typical videopass-through devices, a user can see an outline of the user's phone ifthey hold the phone up to the MR device (e.g., an MR headset orhead-mounted display), but may not be able to see/read the content dueto resolution, distortion, flicker, and/or latency associated with thevideo pass-through. Using the techniques described below, the contentcan be easily viewed by the user on the MR device display (e.g., adisplay of an MR headset or head-mounted display).

VR describes a three-dimensional, sometimes computer-generatedenvironment that can be interacted with in a seemingly real or physicalway. As a user experiencing a VR environment moves in the real world,images rendered in the virtual environment also change, giving the userthe perception that she/he is moving within the virtual environment. Forexample, a user can turn left or right, look up or down, and/or moveforwards or backwards, thus changing the user's point of view of thevirtual environment. The VR content presented to the user can changeaccordingly as the user moves, so that the user's experience is asseamless as in the real world. VR content can include VR video in somecases, which can be captured and rendered at very high quality,potentially providing a truly immersive virtual reality experience. VRapplications include gaming, training, education, video (e.g., sportingevents, movies, shows, among others), online shopping, remote operations(e.g., remote control of a robot, unmanned aerial vehicle, remotesurgery, among others), and/or other applications.

Various devices can be used to render XR content. For example, VRcontent can be rendered and displayed using VR equipment, such as a headmounted display (HMD) or other VR headset, which fully covers a user'seyes during a VR experience. FIG. 1 is a diagram illustrating an exampleof virtual content displayed by a VR headset 102 being worn by a user100. As shown, the user is fully immersed in a VR environment (asillustrated by a VR frame 110).

Because VR equipment can fully cover a user's eyes and in some cases theuser's ears, the user can be unable to interact with other devices(e.g., a mobile phone) and/or the real-world while immersed in thevirtual reality environment. For example, in some cases, the user'sphone or other computing device can be used as a controller for a VRheadset (or other VR device), but there is no ability for the user tointeract with the phone's applications and/or other functionalitieswhile immersed in the VR experience. Further, when using a VR headset,the user can feel vulnerable and out of touch with what is happening inthe real-world environment around them. In some cases, the audioprovided by a VR headset can prevent a user from hearing sounds from theoutside world. When engaged in a VR experience, it would be beneficialto provide an easy way for a user to visually and/or audibly check whatis happening in the real world, without removing the VR headset, andwithout pausing or stopping the virtual reality experience. While a VRheadset can be removed (in which case the VR content may need to bepaused or stopped) so that a user can check the physical surroundingsand/or interact with another device, removal of the headset each timethe user wants to view their phone (or other computing device) or checkthe environment can become tedious and can lead to a poor VR userexperience.

Systems, methods, and computer-readable media are described herein forintegrating computing device and extended reality experiences. As notedabove, examples will be provided herein using VR as an example of a XRexperience. However, one of ordinary skill will appreciate that thetechniques described herein can apply to other types of XR experiences.

According to techniques described herein, a VR device can includemultiple different operating modes. The VR device can include a VRheadset, such as a head mounted display (HMD). The VR device can allow auser to toggle between the multiple modes without pausing or stopping aVR experience being displayed by the VR device. For instance, thevarious modes a user can toggle between include a regular VR mode (shownin FIG. 1, discussed above), a VR mode with a synthetic representationof an actual, real-life computing device and/or content from thecomputing device overlaid on top of the VR content (shown in FIG. 2 andFIG. 3, and discussed below), a VR mode with a peek of the outside worldoverlaid on top of the VR content (shown in FIG. 4 and discussed below),and/or a full switch between viewing VR content and viewing the realworld environment through the VR device (shown in FIG. 5 and discussedbelow). The computing device can include a mobile device (e.g., a mobilephone, a tablet computer, or other mobile device), a wearable device(e.g., a smart watch, a heart rate monitor, or other wearable device),an Internet-of-Things (IoT) device, a remote controller (e.g., atelevision remote, or other remote controller), or other type of device.

The user can toggle between the different modes using one or moreinputs. The input for switching to one or more of the modes can includeany suitable input provided by a user, such as a voice input detectedusing an audio input device (e.g., a microphone) of the VR device and/orthe computing device, a touch input on a touch interface of the VRdevice and/or the computing device, a gesture input detected using oneor more sensors of the VR device and/or the computing device, aselection of a physical button of the VR device or computing device, aphysical button on a VR controller device that can control the VRdevice, selection of an icon or virtual button displayed by the VRdevice or computing device (e.g., a user pressing an icon, a user gazingat an icon, as detected using gaze detection, or other type ofselection), any combination thereof, and/or other input. An example of atouch input that can cause a mode change can include holding down afinger on a touchscreen of the computing device for a threshold amountof time, such as for one second, two seconds, or more seconds and/orwith a certain amount of pressure (e.g., as measured by an accelerometeror other component that can measure pressure or force). Another exampleof a touch input can include a stroke gesture, which can include theuser drawing a circle around the touchscreen with one stroke, and/orsome other stroke gesture. Another example of a touch input can includea strong tap gesture on the touchscreen, which can include a finger tapon the touchscreen with a pressure or force above a force threshold(e.g., as measured by an accelerometer).

In some examples, a movement (e.g., acceleration, speed, etc.) of thecomputing device by more than a threshold amount can be detected andused to trigger one or more of the modes (e.g., the VR mode with thesynthetic representation of an actual computing device and/or contentfrom the computing device overlaid on top of the VR content, the VR modewith a peek of the outside world overlaid on top of the VR content,etc.). In some examples, the VR device can detect (e.g., using anoutward-facing camera of the VR device, using an optical sensor, usingmotion detection, or other suitable technique) that a user has placedthe phone in front of the VR device (e.g., the HMD), in which case theVR device can switch to one or more of the modes (e.g., the VR mode withthe synthetic representation of an actual computing device and/orcontent from the computing device overlaid on top of the VR content, theVR mode with a peek of the outside world overlaid on top of the VRcontent, etc.).

In some examples, the VR device can detect user input using a 6-degreesof freedom (6DOF) input interface. XR devices, such as VR headsets, canconstantly or periodically track their own location in the physicalworld in order to perform the VR functionalities. Such tracking can bereferred to as inside-out 6DOF tracking—inside-out because the devicecan track itself without any external beacons or transmitters and 6DOFbecause the device can track its own position in terms of threerotational vectors (pitch, yaw, roll) and three translational vectors(up/down, left/right, forward/back). One or more sensors can be used todetect user input and/or perform tracking, such as one or more inertialmeasurement units (IMUs), one or more global positioning system (GPS)devices, one or more accelerometers, one or more gyroscopes, anycombination thereof, and/or other sensors. In some cases, tracking canbe accomplished with Visual Inertial Odometry (VIO), which is a processby which visual data (from one or more camera sensors) is fused togetherwith inertial data (from gyroscopes and accelerometers) to measure amoved distance of the device in the physical world. VIO can besimultaneously used to determine a position (localization) of the devicein the world and to map a scene (in the world) associated with thedevice.

FIG. 2 is a diagram illustrating an example of a VR mode of a VR headset102 that allows the VR headset 102 to display a synthetic representation214 of an actual mobile device 204 (a physical device) overlaid on topof a frame 210 of VR content. As illustrated in FIG. 2, the syntheticrepresentation 214 of the mobile device 204 includes a syntheticrepresentation 216 of a display (or screen) of the mobile device 204.The content being displayed by the actual mobile device 204 in the realworld is displayed as virtual content in the synthetic representation216 of the display of the mobile device 204. In this VR mode, the user100 can see a representation of the mobile device 204 display, presentedby a display of the VR headset 102. In some examples, audio content fromthe mobile device 204 can be played by the VR headset 102 whiledisplaying the synthetic representation 214 of the mobile device 204.For example, a voice message can be played when the user selects a voicemessaging application on the mobile device 204. In other examples, avideo (including the visual and audio portions of the video), a videomessage, audio from a web browser, a phone call, streamed music from anetwork location (e.g., from an Internet-based provider), locally-storedmusic (e.g., previously downloaded music), and/or other audio contentcan be played. The VR content can continue to be displayed by the VRheadset 102 as the synthetic representation of the mobile device 204 isdisplayed and/or as the audio content is being played, or can be pausedwhile the synthetic representation of the mobile device 204 is displayedand/or as the audio content is being played by the VR headset 102.

The user can interact with the actual mobile device 204 (withoutremoving the VR headset 102 from the user's head) and can see everythingthat the mobile device 204 displays (and in some cases hear the audiobeing output by the mobile device 204), but inside of the VR headset102. In some cases, the user can interact with the actual mobile device204 using the input interface(s) of the mobile device 204 (e.g., thetouchscreen, push buttons, voice input, touch hover feedback, etc.), andthe change in functionality of the mobile device 204 will be displayedon the synthetic representation 216 of the display of the mobile device204. In some cases, the user can interact with the mobile device 204 byproviding input to the VR headset 102, such as using gesture input, gazeinput, voice input, a VR remote controller, any combination thereof,and/or other suitable input that can be provided to the VR headset 102.In one illustrative example, the user 100 can provide a gesture input tothe VR headset 102 including moving an index finger in a downwarddirection, which upon being detected by the VR headset 102, can causethe content displayed on the synthetic representation 216 of the displayof the mobile device 204 to scroll in a downward direction. In anotherillustrative example, the user 100 can provide a gesture input the VRheadset 102 including moving an index finger in a depth direction awayfrom the VR headset 102 (e.g., as if the user is selecting an icon),which upon being detected by the VR headset 102, can cause an item ofcontent displayed on the synthetic representation 216 of the display ofthe mobile device 204 to be selected. In another illustrative example,the user can interact with the mobile device 204 using a VR remotecontroller as a pointer.

An input (e.g., gestures, gaze, voice, etc.) causing a change in thecontent displayed on the synthetic representation 216 of the display ofthe mobile device 204 can also cause the content displayed by the actualmobile device 204 to change. In one illustrative example, selection of amessaging icon using a gesture detected by the VR headset 102 can causea messaging application to be displayed on the synthetic representation216 of the display of the mobile device 204 and on the display of theactual mobile device 204.

In some examples, user can interact with the actual mobile device 204using touch hover feedback. For instance, the touch hover feedback caninclude the user holding a finger or thumb a certain distance (e.g., 0.5centimeter (cm), 0.75 cm, 1 cm, 1.25 cm, or other suitable distance)from the physical display of the mobile device 204, and a displayelement (e.g., an icon, an item in a list, or other display element inthe synthetic representation 216 of the display of the mobile device204) corresponding to an item the finger or thumb is over can show afocus state in the synthetic representation 216 of the display of themobile device 204 (e.g., the display element can be highlighted, madelarger, and/or otherwise emphasized). Selection of the display elementcan then be performed by the user by selecting the highlighted displayelement (e.g., using a gesture input, gaze input, touch input, voiceinput, and/or other input, as described above). Touch hover feedbackallows interaction with the synthetic representation 214 of the mobiledevice 204 to be more usable by allowing the user to position a fingercorrectly before making a selection. This can be especially helpful inthe example shown in FIG. 2, where the user cannot directly view themobile device 204 display in order to target a selection.

In some implementations, an action notification 215 can be displayedover the frame 210. The action notification 215 indicates to the userthat an action has occurred at the mobile device 204. In some examples,the action notification 215 can include a notification that a messagehas been received by the mobile device 204 (e.g., a new text message hasbeen received, a new email has been received, a notification of anincoming phone to the mobile device 204, a social media post has beennotified by the mobile device 204, an emergency alert has been receivedby the mobile device 204, and/or other messages). In some examples, theaction notification 215 can include the actual message that has beenreceived. In some cases, a user can select the action notification 215to cause the synthetic representation 214 of the mobile device 204 to bedisplayed (if not already displayed).

The synthetic representation 214 of the mobile device 204 can begenerated by the VR headset 102 using various techniques. In oneillustrative example, a user of the VR headset 102 can provide inputindicating the type of the mobile device 204 being used (e.g., the make,model, and version of the mobile device 204, such as a Samsung™ GalaxyS10™), and the VR headset 102 can generate or obtain a virtualrepresentation having the same features and dimensions as that type ofmobile device. In another example, the VR headset 102 can capture one ormore images of the mobile device 204, and the VR headset 102 cangenerate a virtual representation of the mobile device 204 using the oneor more images.

The synthetic representation 214 of the mobile device 204 can bedisplayed in various locations in the VR coordinate system. For example,in some cases, the synthetic representation 214 may be displayed so asto appear in the actual real world location of the phone. In oneillustrative example, if a user is holding the mobile phone in theuser's right hand out in front of the user, the synthetic representation214 of the mobile device 204 can be displayed in the bottom-right cornerof the frame 210. In other examples, the synthetic representation 214can be displayed on an edge of the frame 210 or another location in theframe 210 so that the synthetic representation 214 does not overlap withVR content and/or with a region of highest importance within the frame210 (e.g., a center of the frame 210 corresponding to the field-of-viewof the VR user). In some examples, the location at which the syntheticrepresentation 214 of the mobile device 204 is displayed can be userselectable. For example, a user can indicate that the syntheticrepresentation 214 is to be displayed in the top-left corner of theframes displayed by the VR headset 102.

In some cases, the content being displayed and/or audio content beingoutput by the actual mobile device 204 can be transmitted or streamed tothe VR headset 102, and the VR headset 102 can display the content inthe synthetic representation 216 of the display of the mobile device204. In some cases, the content being displayed by the mobile device 204can be mirrored (e.g., using screen mirroring, such as Miracast usingthe peer-to-peer WiFi Direct standard or AirPlay) on the display of theVR headset 102.

FIG. 6 is a diagram illustrating an example of components of an XRdevice, such as a VR device (e.g., of VR headset 102 or other VRdevice), an MR device, and/or an AR device. The components operate toenable content being displayed and/or audio content being output by anactual mobile device 204 (e.g., mobile device 204) to be displayed bythe XR device. The XR device can include one or more applications 632.An application can register for Runtime services (e.g., rendering and/orother Runtime services) provided by Runtime engine 634. The Runtimeengine 634 can handle rendering for each display or graphic layer thatcan be displayed by the XR device. The Runtime engine 634 can alsoprocess head tracking pose and input events for each layer (e.g.,controller, hand gesture, voice, and/or other input events). In somecases, the Runtime engine 634 can be part of an application from the oneor more applications 632, part of the operating system (OS) 638, or canbe combined with the compositor engine 636 in cases when the Runtimeengine 634 coordinates XR activities which may include composition.

The compositor engine 636 can weave multiple graphic layers (e.g.,windows, tiles, among others) together for viewing within an XR (e.g., aVR session). The compositor engine 636 can also perform processesXR-related processes, such as timewarp, spacewarp, among others. In somecases, the compositor engine 636 can be part of the Runtime engine 634,as described above, or can be part of the OS 638. The OS 638 can be anysuitable OS, such as Android™ or other suitable OS.

FIG. 7 is a diagram illustrating an example of different graphic layersthat can be handled by Runtime engine 634. For example, the Layer Ashown in FIG. 7 corresponds to the content being displayed by a mobiledevice (e.g., mobile device 204), and the Layer B corresponds to the XRcontent being displayed by the XR device (e.g., VR headset 102). VRcontent will be used as an example of XR content. However, one ofordinary skill will appreciate that the XR content can include MRcontent, AR content, and/or other content. The OS 638 (e.g., Android orother OS) can assign a mobile device user interface (UI) to one display(e.g., a primary display) and a VR view can be assigned to anotherdisplay (e.g., a secondary display). The Runtime engine 634 can renderthe primary display assigned to the mobile device content as a layer(e.g., Layer A in FIG. 7), which can be viewed as a video stream on asecondary VR display rendered as another layer (e.g., Layer B in FIG.7). The compositor engine 636 can weave together each layer so that themobile device view layer (Layer A) appears in the foreground and the VRview layer (Layer B) runs in the background behind Layer A.

The Runtime engine 634 can handle input events (e.g., a controller, handgesture, voice, etc.) from a layered VR session. For example, an inputevent selecting an item on the mobile device layer (Layer A) can causethe Runtime engine 634 to map the input event to a touch event, which isthen passed back to the mobile device to be processed. In anotherexample, an input event selecting an item on the VR layer (Layer B) cancause the Runtime engine 634 to pass the input event to the VR device tobe processed.

FIG. 8 is a diagram illustrating another example of components of avirtual reality device. As shown, an XR application 832 controls an XRapplication user interface (UI) 833 of the XR application 832. A mobiledevice 804 controls a mobile device UI 835 of the mobile device 804.User input provided by a user to the mobile device 804 can be providedto the mobile device UI 835 by the mobile device 804. Similarly, userinput provided by a user to the XR device can be provided to the XRapplication UI 833 by the XR device. Users can interact with the mobiledevice UI 835 and/or the XR application UI 833 using input devices, suchas controllers, pointers, gaze, hand tracking input, voice, amongothers. An XR Runtime engine 834 can process the user inputs and canhandle rendering for each graphic layer that can be displayed in the XRscene user interface 844 by the one or more XR device displays 846 ofthe XR device. The XR scene user interface 844 includes all contentbeing displayed by the XR device, including the content of the XRapplication UI 833 and/or UIs of other XR applications on the XR device.

In some cases, mobile devices (e.g., mobile phones) can have multipledisplays, which may support UI interactions (e.g., smartphone UIinteractions) and display concurrently with XR application interactionsand display. Content displayed by the mobile device 804 (e.g., a mobiledevice UI 835) on the mobile device display 84 can be represented in anXR scene user interface 844 in its entirety (e.g., all content displayedby a mobile device can be displayed with the XR content) or can berepresented in component form (e.g., a single application UI or UIfragment of a single application UI can be displayed with the XRcontent).

The mobile device UI 835 can be visible on the mobile device display842, on the one or more XR device displays 846, on both the mobiledevice display 842 and the one or more XR device displays 846, or onnone of the displays. In some cases, the XR Runtime engine 834 canreside within or outside of the XR application 832. In some cases, theXR Runtime engine 834 can reside within or outside an OS, as describedabove. In some cases, the XR Runtime engine 834 can be co-resident onthe mobile device and the XR device. In some cases, the XR Runtimeengine 834 can reside on a connected device, such as an all-in-one VRheadset that incorporates a mobile device. The connection between the XRdevice and the connected device can be wired or wireless (e.g., usingWiFi, Bluetooth™, Zigbee™, or other wireless connection.

Using the mode shown in FIG. 2, the user 100 can view the mobile device204 presented on a display of the VR headset 102 (and in some cases canhear audio content from the mobile device 204) while still viewing theVR content, and can interact with all of the functionalities of themobile device 204 as if the user was interacting directly with themobile device 204. In some cases, the VR content can continue playingwhile the user 100 is interacting with the mobile device 204. In somecases, the VR headset 102 can provide an option for the user 100 topause the VR content while interacting with the mobile device 204. Ineither case, the user can interact with the mobile device 204, withoutthe need to remove the VR headset 102.

FIG. 3 is a diagram illustrating an example of a VR mode of a VR headset102 that allows the VR headset 102 to display content from the mobiledevice 204 overlaid on top of a frame 310 of VR content. The contentbeing displayed by the actual mobile device 204 in the real world isdisplayed as virtual content 317 over the frame 310 of virtual content.In the VR mode, the user 100 can see what is being displayed by themobile device 204, but without displaying a synthetic representation ofthe mobile device 204. The virtual content 317 can be displayed invarious locations in the VR coordinate system. For example, in somecases, the virtual content 317 may be displayed so as to appear in theactual real world location of the phone. In other examples, the virtualcontent 317 can be displayed on an edge of the frame 310 (e.g., as shownin FIG. 3) or otherwise located in the frame 310 so that the virtualcontent 317 does not collide with VR content and/or with a region ofhighest importance (e.g., a center of the frame 310 corresponding to thefield-of-view of the VR user). In some examples, the location at whichthe virtual content 317 is displayed can be user selectable.

In some examples, audio content from the mobile device 204 can be playedby the VR headset 102 while displaying the content from the mobiledevice 204. The VR content can continue to be displayed by the VRheadset 102 as the content from the mobile device 204 is displayedand/or as the audio content is being played, or can be paused while thecontent from the mobile device 204 is displayed and/or as the audiocontent is being played by the VR headset 102.

Similar to that described with respect to the mode shown in FIG. 2, theuser can interact with the actual mobile device 204 (without removingthe VR headset 102) and can see everything that is displayed by thephone, but inside of the VR headset 102. The user can interact with theactual mobile device 204 using the input interface(s) of the mobiledevice 204 and/or can interact with the mobile device 204 by providinginput to the VR headset 102, similar to that described above withrespect to FIG. 2.

The synthetic representation 214 of the mobile device 204 can begenerated by the VR headset 102 based on input provided by a userindicating the type of the mobile device 204 being used, based on one ormore images of the mobile device 204 captured by the VR headset 102,and/or using any other suitable technique. The content being displayedby the actual mobile device 204 can be transmitted or streamed to the VRheadset 102, can be mirrored (e.g., using screen mirroring) on thedisplay of the VR headset 102, or provided to the VR headset 102 usingany other suitable technique.

Similar to that described with respect to FIG. 2, an action notification315 can be displayed over the frame 310. The action notification 315 canindicate that an action has occurred at the mobile device 204, such as amessage or phone call being received. In some examples, the actionnotification 315 can include the actual message that has been received.A user can select the action notification 315 to cause the virtualcontent 317 from the mobile device 204 to be displayed (if not alreadybeing displayed). For example, the messaging application from the mobiledevice 204 can be displayed on the VR device 102 in response toselection of the action notification 315.

Using the mode shown in FIG. 3, the user 100 can view content from themobile device 204 presented on a display of the VR headset 102 whilestill viewing the VR content, and can interact with all of thefunctionalities of the mobile device 204 as if the user was interactingdirectly with the mobile device 204.

FIG. 4 is a diagram illustrating an example of a VR mode of a VR headset102 that allows the VR headset 102 to provide a peek of the outsideworld by displaying one or more images (including image 418) captured bythe mobile device 204 as an overlay overlaid on top of a frame 410 of VRcontent. The one or more images can be captured by a camera or multiplecameras of the mobile device 204, and can include one or more stillimages or a video including multiple consecutive images. In some cases,the camera of the mobile device 204 used to capture the one or moreimages can be the back-facing camera (located on the back side of themobile device 204), and/or another camera of the mobile device 204(e.g., a front-facing camera, a front-facing camera and a back-facingcamera, multiple front-facing and/or back-facing cameras, or the like).The mobile device 204 can transmit the one or more images to the VRheadset 102, which can then display the one or more images on thedisplay of the VR headset 102. In some cases, audio from the microphoneof the mobile device 204 can be provided to the VR headset 102, and theVR headset 102 can output the audio using one or more speakers of the VRheadset 102. Providing audio from the microphone of the mobile device204 can provide the user with more information to allow the user toperceive what is going on in the real world without having to remove theheadset.

The example shown in FIG. 4 includes the image 418 displayed as part ofa synthetic representation 414 of the mobile device 204. In someimplementations, the one or more images captured by the mobile device204 (including image 418) is displayed over the virtual content, inwhich case the synthetic representation 414 of the mobile device 204 isnot displayed by the VR headset 102. In some implementations, the one ormore images can be fully displayed on the display of the VR headset 102(e.g., in full-screen mode), in which case no VR content is displayed bythe VR headset 102 while the one or more images are displayed. In someimplementations, the user 100 can control how much of the VR headset 102display is used for displaying the one or more images.

Using the mode shown in FIG. 4, the user 100 can see a view of thereal-world environment from a camera of the mobile device 204, presentedon a display of the VR headset 102 while still viewing the VR content.As noted above, the user can also be provided with audio from amicrophone of the mobile device 204, which can allow the user to hearany sounds that are occurring in the real-world environment. In somecases, the VR headset 102 can provide an option for the user 100 topause the VR content while viewing the one or more images from themobile device 204. In either case, the user can see a view of thereal-world environment (and in some cases can hear what is happening inthe real-world environment), without the need to remove the VR headset102.

FIG. 5 is a diagram illustrating an example of a VR mode of a VR headset102 that allows the VR headset 102 to switch between the display ofvirtual VR content and the display of a real world. For example, frames(including a frame 510) of VR content can be displayed by the VR headset102, which include a scene of a virtual environment based on the pose ofthe user. In response to user input, the VR headset 102 can switch to areal-world display mode and, once in the real-world display mode, candisplay a view (including frame 520) of the real-world environment basedon the pose of the user.

In some cases, an external camera that is part of the VR headset 102 canbe used to capture images of the real-world environment. The capturedimages can be displayed by the display of the VR headset 102 when theuser switches to the real-world display mode. The captured images caninclude one or more still images, or a video including multipleconsecutive images. The still image(s) or the video can be displayeduntil the user provides input instructing the VR headset 102 to switchback to the full VR mode where VR content is displayed without the oneor more still images.

Using the mode shown in FIG. 5, the user 100 can toggle between a viewof the VR world and the real-world environment where the user islocated, with no need to take off the VR headset 102 in order tointeract with the real-world environment. For example, the user 100 canbe provided with a full view of a room the user 100 is sitting in, sothat the user 100 can interact with objects in the room while stillwearing the VR headset 102.

Examples of processes performed using the techniques described hereinwill now be described. FIG. 9 is a flowchart illustrating an example ofa process 900 for displaying virtual content. At block 902, the process900 includes displaying, by a first device, virtual reality content. Thefirst device can include a virtual reality (VR) headset (e.g., an HMD orother type of VR headset), such as the VR headset 102 shown in FIG.1-FIG. 5.

At block 904, the process 900 includes generating, by the first device,a synthetic representation of a second device. The second device caninclude a mobile device, such as the mobile device 204 shown in FIG.1-FIG. 4. At block 906, the process 900 includes displaying, by thefirst device, the synthetic representation of the second device with thevirtual content. In some examples, the synthetic representation of thesecond device is overlaid over the virtual content. For instance, usingFIG. 2 as illustrative example, the synthetic representation 214 of themobile device 204 is displayed along with a virtual content frame 210.

The synthetic representation of the second device (e.g., the syntheticrepresentation 214 shown in FIG. 2) can include a syntheticrepresentation of a display of the second device. Content displayed bythe second device is displayed in the synthetic representation of thedisplay of the second device. For instance, as shown in FIG. 2, thesynthetic representation 214 of the mobile device 204 also includes asynthetic representation 216 of the display of the mobile device 204,and content being displayed by the mobile device 204 is shown in thesynthetic representation 216 of the display of the mobile device 204.

In some cases, the process 900 can include displaying, based on anindication of one or more inputs processed by the second device, achange in the synthetic representation of the second device. The changein the synthetic representation of the second device represents a changein functionality of the second device. For example, a user can interactwith the second device (e.g., a mobile device) in order to cause thesecond device to perform one or more functions (e.g., send a textmessage using a messaging application, post a message to a socialnetworking account using a social networking application, send an email,among others). The change in content displayed by the second devicebased on the change in functionality of the second device can also bedisplayed as the change in the synthetic representation of the seconddevice representing the change in functionality of the second device.

In some cases, the process 900 can include displaying, based on inputreceived by the first device, a change in the synthetic representationof the second device. The change in the synthetic representation of thesecond device represents a change in functionality of the second device.For example, a user can interact with the first device (e.g., a VRheadset) or the second device in order to cause the second device toperform one or more functions (e.g., send a text message using amessaging application, post a message to a social networking accountusing a social networking application, send an email, among others). Theuser can interact with the second device to cause the second device toperform the one or more functions using any suitable input, such asthose described above (e.g., using the input interface(s) of the mobiledevice, by providing input to the VR headset, using touch hoverfeedback, or other suitable input). The change in content displayed bythe second device based on the change in functionality of the seconddevice can also be displayed as the change in the syntheticrepresentation of the second device representing the change infunctionality of the second device.

In some examples, the process 900 includes detecting a trigger, anddisplaying the synthetic representation of the second device (along withcontent displayed by the second device) in response to the trigger. Insome cases, the trigger is based on information received from the seconddevice. In one illustrative example, the information received from thesecond device includes a notification that a message has been receivedby the second device (e.g., that a new text message or email has beenreceived), an input received by the second device (e.g., a user inputprovided to the second device), and/or other notification. In somecases, the trigger is based on the second device being moved by morethan a threshold amount. For example, an accelerometer or other sensorof the second device can provide information indicating an amount ofmovement of the second device. In some cases, the trigger is based ondetection of a gaze of a user directed toward the second device, asdetected by the first device (e.g., using 6DOF tracking). In some cases,the trigger is based on an amount of time. For instance, the syntheticrepresentation of the second device (along with content displayed by thesecond device) can be displayed by the first device with the virtualcontent at every threshold interval of time (e.g., every five minutes,every ten minutes, or other suitable time period) if no other triggerhas been received. In some cases, the trigger is based on selection ofan icon displayed with the virtual content. For example, the user canselect the icon using a gesture, eye gaze, voice command, and/or otherinput. In some examples, trigger is based on voice input provided by auser. For instance, a user can say aloud “show my mobile device” orother phrase, which can be received as input by the first device andinterpreted as a trigger causing the synthetic representation of thesecond device to be displayed. In some examples, the trigger is based ona user lifting the mobile device and/or unlocking the mobile device(e.g., via fingerprint input, by pressing hard button, via facerecognition or face identification (ID), or using another mechanism forunlocking a device).

The synthetic representation of the second device can be removed fromdisplay by the first device based on various removal triggers. Forexample, a user can provide a touch input (e.g., a swipe input on a userinterface), a voice input, a gaze input, a gesture input, and/or otherinput to the first device or the second device, which can be interpretedby the first device as a removal trigger that triggers the first deviceto remove the synthetic representation of the second device from beingdisplayed. In another example, a removal trigger can be based on a usermoving the second device to a particular point or using a particularmotion (e.g., by lowering the device, as detected by an accelerometer orother sensor of the second device), causing the first device to removethe synthetic representation of the second device from display. Inanother example, the synthetic representation of the second device canbe removed based on an amount of time since a last interaction with thesecond device. For example, if a threshold amount of time has passedsince an input has been received from the second device or since theuser has interacted with the synthetic representation of the seconddevice, the synthetic representation can be removed from display. As aresult of removing the synthetic representation of the second devicefrom display, the virtual content will be displayed without thesynthetic representation of the second device.

FIG. 10 is a flowchart illustrating another example of a process 1000for displaying virtual content. At block 1002, the process 1000 includesdisplaying, by a first device, virtual reality content. The first devicecan include a virtual reality (VR) headset (e.g., an HMD or other typeof VR headset), such as the VR headset 102 shown in FIG. 1-FIG. 5.

At block 1004, the process 1000 includes obtaining, by the first device,content displayed by a second device. The second device can include amobile device, such as the mobile device 204 shown in FIG. 1-FIG. 4. Atblock 1006, the process 1000 includes displaying, by the first device,the content with the virtual content. In some cases, the contentincludes information from at least one application executed by thesecond device, such as an email application, a messaging application, asocial networking application, a business application, a mapsapplication, and/or other suitable application. In some examples, thecontent is overlaid over the virtual content. For instance, using FIG. 3as illustrative example, the content displayed by the mobile device 204is displayed as virtual content 317 over the frame 310 of virtualcontent. In some examples, a synthetic representation of the seconddevice is overlaid over the virtual content, and the content can bedisplayed with the synthetic representation of the second device (e.g.,as shown in FIG. 2).

In some cases, the process 1000 can include displaying, based on anindication of one or more inputs processed by the second device, achange in content of the second device. The change in the contentrepresents a change in functionality of the second device. For example,a user can interact with the second device (e.g., a mobile device) inorder to cause the second device to perform one or more functions (e.g.,send a text message using a messaging application, post a message to asocial networking account using a social networking application, send anemail, among others). The change in content displayed by the seconddevice based on the change in functionality of the second device canalso be displayed as the change in the content representing the changein functionality of the second device.

In some examples, the process 1000 can include displaying, based oninput received by the first device, a change in content of the seconddevice. The change in content of the second device represents a changein functionality of the second device. For example, a user can interactwith the first device (e.g., a VR headset) in order to cause the seconddevice to perform one or more functions (e.g., send a text message usinga messaging application, post a message to a social networking accountusing a social networking application, send an email, among others). Thechange in content displayed by the second device based on the change infunctionality of the second device can also be displayed as the changein content of the second device representing the change in functionalityof the second device.

In some examples, the process 1000 includes detecting a trigger, anddisplaying the content with the virtual content in response to thetrigger. In some cases, the trigger is based on information receivedfrom the second device. In one illustrative example, the informationreceived from the second device includes a notification that a messagehas been received by the second device (e.g., that a new text message oremail has been received), an input received by the second device (e.g.,a user input provided to the second device), and/or other notification.In some cases, the trigger is based on the second device being moved bymore than a threshold amount. For example, an accelerometer or othersensor of the second device can provide information indicating an amountof movement of the second device. In some cases, the trigger is based ondetection of a gaze of a user directed toward the second device, asdetected by the first device. In some cases, the trigger is based on anamount of time. For instance, content from the second device can bedisplayed by the first device with the virtual content at everythreshold interval of time (e.g., every five minutes, every ten minutes,or other suitable time period) if no other trigger has been received. Insome cases, the trigger is based on selection of an icon displayed withthe virtual content. For example, the user can select the icon using agesture, eye gaze, voice command, and/or other input. In some examples,trigger is based on voice input provided by a user. For instance, a usercan say aloud “show my mobile device” or other phrase, which can bereceived as input by the first device and interpreted as a triggercausing the content displayed by the second device to be displayed. Insome examples, the trigger is based on a user lifting the mobile deviceand/or unlocking the mobile device (e.g., via fingerprint input, bypressing hard button, via face recognition or face identification (ID),or using another mechanism for unlocking a device).

The content displayed by the second device can be removed from displayby the first device based on various removal triggers. For example, auser can provide a touch input (e.g., a swipe input on a userinterface), a voice input, a gaze input, a gesture input, and/or otherinput to the first device or the second device, which can be interpretedby the first device as a removal trigger that triggers the first deviceto remove the content displayed by the second device from beingdisplayed on the first device. In another example, a removal trigger canbe based on a user moving the second device to a particular point orusing a particular motion (e.g., by lowering the device, as detected byan accelerometer or other sensor of the second device). In anotherexample, the content displayed by the second device can be removed basedon an amount of time since a last interaction with the second device.For example, if a threshold amount of time has passed since an input hasbeen received from the second device or since the user has interactedwith the content from the second device being displayed by the firstdevice, the content displayed by the second device can be removed fromdisplay on the first device. As a result of removing the contentdisplayed by the second device from display, the virtual content will bedisplayed without the content displayed by the second device.

FIG. 11 is a flowchart illustrating another example of a process 1100for displaying virtual content. At block 1102, the process 1100 includesdisplaying, by a first device, virtual reality content. The first devicecan include a virtual reality (VR) headset (e.g., an HMD or other typeof VR headset), such as the VR headset 102 shown in FIG. 1-FIG. 5.

At block 1104, the process 1100 includes obtaining, by the first device,one or more images captured by a camera of a second device. The seconddevice can include a mobile device, such as the mobile device 204 shownin FIG. 1-FIG. 4. At block 1106, the process 1100 includes displaying,by the first device, the one or more images with the virtual content. Insome examples, the one or more images are overlaid over the virtualcontent. For instance, using FIG. 4 as illustrative example, an image418 captured by the mobile device 204 is displayed along with a virtualcontent frame 410. In some examples, the one or more images include avideo captured by the camera of the second device. In some examples, theone or more images include at least one still image captured by thecamera of the second device.

In some implementations, the process 1100 includes displaying, using thefirst device, a synthetic representation of the second device with thevirtual content. The one or more images can be displayed with thevirtual content as part of the synthetic representation of the seconddevice. For instance, the one or more images can be displayed within asynthetic representation of a display of the second device. In oneexample using FIG. 4 for illustrative purposes, a syntheticrepresentation 414 of the mobile device 204 is shown with the image 418displayed in a synthetic representation of a display representing theactual display of the mobile device 204.

In some examples, the processes 900, 1000, and/or 1100 may be performedby a computing device or apparatus, such as a computing device havingthe computing device architecture 1200 shown in FIG. 12. In oneillustrative example, the computing device (e.g., performing theprocesses 900, 1000, and/or 1100) can include an extended realitydisplay device, such as a head mounted display (HMD) or other type of XRheadset.

In some cases, the computing device or apparatus may include an inputdevice, an output device, one or more processors, one or moremicroprocessors, one or more microcomputers, and/or other component(s)that is/are configured to carry out the steps of processes 900, 1000,and/or 1100. The components of the computing device (e.g., the one ormore processors, one or more microprocessors, one or moremicrocomputers, and/or other component) can be implemented in circuitry.For example, the components can include and/or can be implemented usingelectronic circuits or other electronic hardware, which can include oneor more programmable electronic circuits (e.g., microprocessors,graphics processing units (GPUs), digital signal processors (DSPs),central processing units (CPUs), and/or other suitable electroniccircuits), and/or can include and/or be implemented using computersoftware, firmware, or any combination thereof, to perform the variousoperations described herein. The computing device may further include adisplay (as an example of the output device or in addition to the outputdevice), a network interface configured to communicate and/or receivethe data, any combination thereof, and/or other component(s). Thenetwork interface may be configured to communicate and/or receiveInternet Protocol (IP) based data or other type of data.

The processes 900, 1000, and/or 1100 are illustrated as logical flowdiagrams, the operation of which represent a sequence of operations thatcan be implemented in hardware, computer instructions, or a combinationthereof. In the context of computer instructions, the operationsrepresent computer-executable instructions stored on one or morecomputer-readable storage media that, when executed by one or moreprocessors, perform the recited operations. Generally,computer-executable instructions include routines, programs, objects,components, data structures, and the like that perform particularfunctions or implement particular data types. The order in which theoperations are described is not intended to be construed as alimitation, and any number of the described operations can be combinedin any order and/or in parallel to implement the processes.

Additionally, the processes 900, 1000, and/or 1100 may be performedunder the control of one or more computer systems configured withexecutable instructions and may be implemented as code (e.g., executableinstructions, one or more computer programs, or one or moreapplications) executing collectively on one or more processors, byhardware, or combinations thereof. As noted above, the code may bestored on a computer-readable or machine-readable storage medium, forexample, in the form of a computer program comprising a plurality ofinstructions executable by one or more processors. The computer-readableor machine-readable storage medium may be non-transitory.

FIG. 12 illustrates an example computing device architecture 1200 of anexample computing device which can implement the various techniquesdescribed herein. The components of computing device architecture 1200are shown in electrical communication with each other using connection1205, such as a bus. The example computing device architecture 1200includes a processing unit (CPU or processor) 1210 and computing deviceconnection 1205 that couples various computing device componentsincluding computing device memory 1215, such as read only memory (ROM)1220 and random access memory (RAM) 1225, to processor 1210.

Computing device architecture 1200 can include a cache of high-speedmemory connected directly with, in close proximity to, or integrated aspart of processor 1210. Computing device architecture 1200 can copy datafrom memory 1215 and/or the storage device 1230 to cache 1212 for quickaccess by processor 1210. In this way, the cache can provide aperformance boost that avoids processor 1210 delays while waiting fordata. These and other modules can control or be configured to controlprocessor 1210 to perform various actions. Other computing device memory1215 may be available for use as well. Memory 1215 can include multipledifferent types of memory with different performance characteristics.Processor 1210 can include any general purpose processor and a hardwareor software service, such as service 1 1232, service 2 1234, and service3 1236 stored in storage device 1230, configured to control processor1210 as well as a special-purpose processor where software instructionsare incorporated into the processor design. Processor 1210 may be aself-contained system, containing multiple cores or processors, a bus,memory controller, cache, etc. A multi-core processor may be symmetricor asymmetric.

To enable user interaction with the computing device architecture 1200,input device 1245 can represent any number of input mechanisms, such asa microphone for speech, a touch-sensitive screen for gesture orgraphical input, keyboard, mouse, motion input, speech and so forth.Output device 1235 can also be one or more of a number of outputmechanisms known to those of skill in the art, such as a display,projector, television, speaker device, etc. In some instances,multimodal computing devices can enable a user to provide multiple typesof input to communicate with computing device architecture 1200.Communication interface 1240 can generally govern and manage the userinput and computing device output. There is no restriction on operatingon any particular hardware arrangement and therefore the basic featureshere may easily be substituted for improved hardware or firmwarearrangements as they are developed.

Storage device 1230 is a non-volatile memory and can be a hard disk orother types of computer readable media which can store data that areaccessible by a computer, such as magnetic cassettes, flash memorycards, solid state memory devices, digital versatile disks, cartridges,random access memories (RAMs) 1225, read only memory (ROM) 1220, andhybrids thereof. Storage device 1230 can include services 1232, 1234,1236 for controlling processor 1210. Other hardware or software modulesare contemplated. Storage device 1230 can be connected to the computingdevice connection 1205. In one aspect, a hardware module that performs aparticular function can include the software component stored in acomputer-readable medium in connection with the necessary hardwarecomponents, such as processor 1210, connection 1205, output device 1235,and so forth, to carry out the function.

The techniques of this disclosure are not necessarily limited towireless applications or settings. The techniques may be applied tovideo coding in support of any of a variety of multimedia applications,such as over-the-air television broadcasts, cable televisiontransmissions, satellite television transmissions, Internet streamingvideo transmissions, such as dynamic adaptive streaming over HTTP(DASH), digital video that is encoded onto a data storage medium,decoding of digital video stored on a data storage medium, or otherapplications. In some examples, system may be configured to supportone-way or two-way video transmission to support applications such asvideo streaming, video playback, video broadcasting, and/or videotelephony.

As used herein, the term “computer-readable medium” includes, but is notlimited to, portable or non-portable storage devices, optical storagedevices, and various other mediums capable of storing, containing, orcarrying instruction(s) and/or data. A computer-readable medium mayinclude a non-transitory medium in which data can be stored and thatdoes not include carrier waves and/or transitory electronic signalspropagating wirelessly or over wired connections. Examples of anon-transitory medium may include, but are not limited to, a magneticdisk or tape, optical storage media such as compact disk (CD) or digitalversatile disk (DVD), flash memory, memory or memory devices. Acomputer-readable medium may have stored thereon code and/ormachine-executable instructions that may represent a procedure, afunction, a subprogram, a program, a routine, a subroutine, a module, asoftware package, a class, or any combination of instructions, datastructures, or program statements. A code segment may be coupled toanother code segment or a hardware circuit by passing and/or receivinginformation, data, arguments, parameters, or memory contents.Information, arguments, parameters, data, etc. may be passed, forwarded,or transmitted via any suitable means including memory sharing, messagepassing, token passing, network transmission, or the like.

In some embodiments the computer-readable storage devices, mediums, andmemories can include a cable or wireless signal containing a bit streamand the like. However, when mentioned, non-transitory computer-readablestorage media expressly exclude media such as energy, carrier signals,electromagnetic waves, and signals per se.

Specific details are provided in the description above to provide athorough understanding of the embodiments and examples provided herein.However, it will be understood by one of ordinary skill in the art thatthe embodiments may be practiced without these specific details. Forclarity of explanation, in some instances the present technology may bepresented as including individual functional blocks including functionalblocks comprising devices, device components, steps or routines in amethod embodied in software, or combinations of hardware and software.Additional components may be used other than those shown in the figuresand/or described herein. For example, circuits, systems, networks,processes, and other components may be shown as components in blockdiagram form in order not to obscure the embodiments in unnecessarydetail. In other instances, well-known circuits, processes, algorithms,structures, and techniques may be shown without unnecessary detail inorder to avoid obscuring the embodiments.

Individual embodiments may be described above as a process or methodwhich is depicted as a flowchart, a flow diagram, a data flow diagram, astructure diagram, or a block diagram. Although a flowchart may describethe operations as a sequential process, many of the operations can beperformed in parallel or concurrently. In addition, the order of theoperations may be re-arranged. A process is terminated when itsoperations are completed, but could have additional steps not includedin a figure. A process may correspond to a method, a function, aprocedure, a subroutine, a subprogram, etc. When a process correspondsto a function, its termination can correspond to a return of thefunction to the calling function or the main function.

Processes and methods according to the above-described examples can beimplemented using computer-executable instructions that are stored orotherwise available from computer-readable media. Such instructions caninclude, for example, instructions and data which cause or otherwiseconfigure a general purpose computer, special purpose computer, or aprocessing device to perform a certain function or group of functions.Portions of computer resources used can be accessible over a network.The computer executable instructions may be, for example, binaries,intermediate format instructions such as assembly language, firmware,source code, etc. Examples of computer-readable media that may be usedto store instructions, information used, and/or information createdduring methods according to described examples include magnetic oroptical disks, flash memory, USB devices provided with non-volatilememory, networked storage devices, and so on.

Devices implementing processes and methods according to thesedisclosures can include hardware, software, firmware, middleware,microcode, hardware description languages, or any combination thereof,and can take any of a variety of form factors. When implemented insoftware, firmware, middleware, or microcode, the program code or codesegments to perform the necessary tasks (e.g., a computer-programproduct) may be stored in a computer-readable or machine-readablemedium. A processor(s) may perform the necessary tasks. Typical examplesof form factors include laptops, smart phones, mobile phones, tabletdevices or other small form factor personal computers, personal digitalassistants, rackmount devices, standalone devices, and so on.Functionality described herein also can be embodied in peripherals oradd-in cards. Such functionality can also be implemented on a circuitboard among different chips or different processes executing in a singledevice, by way of further example.

The instructions, media for conveying such instructions, computingresources for executing them, and other structures for supporting suchcomputing resources are example means for providing the functionsdescribed in the disclosure.

In the foregoing description, aspects of the application are describedwith reference to specific embodiments thereof, but those skilled in theart will recognize that the application is not limited thereto. Thus,while illustrative embodiments of the application have been described indetail herein, it is to be understood that the inventive concepts may beotherwise variously embodied and employed, and that the appended claimsare intended to be construed to include such variations, except aslimited by the prior art. Various features and aspects of theabove-described application may be used individually or jointly.Further, embodiments can be utilized in any number of environments andapplications beyond those described herein without departing from thebroader spirit and scope of the specification. The specification anddrawings are, accordingly, to be regarded as illustrative rather thanrestrictive. For the purposes of illustration, methods were described ina particular order. It should be appreciated that in alternateembodiments, the methods may be performed in a different order than thatdescribed.

One of ordinary skill will appreciate that the less than (“<”) andgreater than (“>”) symbols or terminology used herein can be replacedwith less than or equal to (“≤”) and greater than or equal to (“≥”)symbols, respectively, without departing from the scope of thisdescription.

Where components are described as being “configured to” perform certainoperations, such configuration can be accomplished, for example, bydesigning electronic circuits or other hardware to perform theoperation, by programming programmable electronic circuits (e.g.,microprocessors, or other suitable electronic circuits) to perform theoperation, or any combination thereof.

The phrase “coupled to” refers to any component that is physicallyconnected to another component either directly or indirectly, and/or anycomponent that is in communication with another component (e.g.,connected to the other component over a wired or wireless connection,and/or other suitable communication interface) either directly orindirectly.

Claim language or other language reciting “at least one of” a set and/or“one or more” of a set indicates that one member of the set or multiplemembers of the set (in any combination) satisfy the claim. For example,claim language reciting “at least one of A and B” means A, B, or A andB. In another example, claim language reciting “at least one of A, B,and C” means A, B, C, or A and B, or A and C, or B and C, or A and B andC. The language “at least one of” a set and/or “one or more” of a setdoes not limit the set to the items listed in the set. For example,claim language reciting “at least one of A and B” can mean A, B, or Aand B, and can additionally include items not listed in the set of A andB.

The various illustrative logical blocks, modules, circuits, andalgorithm steps described in connection with the embodiments disclosedherein may be implemented as electronic hardware, computer software,firmware, or combinations thereof. To clearly illustrate thisinterchangeability of hardware and software, various illustrativecomponents, blocks, modules, circuits, and steps have been describedabove generally in terms of their functionality. Whether suchfunctionality is implemented as hardware or software depends upon theparticular application and design constraints imposed on the overallsystem. Skilled artisans may implement the described functionality invarying ways for each particular application, but such implementationdecisions should not be interpreted as causing a departure from thescope of the present application.

The techniques described herein may also be implemented in electronichardware, computer software, firmware, or any combination thereof. Suchtechniques may be implemented in any of a variety of devices such asgeneral purposes computers, wireless communication device handsets, orintegrated circuit devices having multiple uses including application inwireless communication device handsets and other devices. Any featuresdescribed as modules or components may be implemented together in anintegrated logic device or separately as discrete but interoperablelogic devices. If implemented in software, the techniques may berealized at least in part by a computer-readable data storage mediumcomprising program code including instructions that, when executed,performs one or more of the methods described above. Thecomputer-readable data storage medium may form part of a computerprogram product, which may include packaging materials. Thecomputer-readable medium may comprise memory or data storage media, suchas random access memory (RAM) such as synchronous dynamic random accessmemory (SDRAM), read-only memory (ROM), non-volatile random accessmemory (NVRAM), electrically erasable programmable read-only memory(EEPROM), FLASH memory, magnetic or optical data storage media, and thelike. The techniques additionally, or alternatively, may be realized atleast in part by a computer-readable communication medium that carriesor communicates program code in the form of instructions or datastructures and that can be accessed, read, and/or executed by acomputer, such as propagated signals or waves.

The program code may be executed by a processor, which may include oneor more processors, such as one or more digital signal processors(DSPs), general purpose microprocessors, an application specificintegrated circuits (ASICs), field programmable logic arrays (FPGAs), orother equivalent integrated or discrete logic circuitry. Such aprocessor may be configured to perform any of the techniques describedin this disclosure. A general purpose processor may be a microprocessor;but in the alternative, the processor may be any conventional processor,controller, microcontroller, or state machine. A processor may also beimplemented as a combination of computing devices, e.g., a combinationof a DSP and a microprocessor, a plurality of microprocessors, one ormore microprocessors in conjunction with a DSP core, or any other suchconfiguration. Accordingly, the term “processor,” as used herein mayrefer to any of the foregoing structure, any combination of theforegoing structure, or any other structure or apparatus suitable forimplementation of the techniques described herein. In addition, in someaspects, the functionality described herein may be provided withindedicated software modules or hardware modules configured for encodingand decoding, or incorporated in a combined video encoder-decoder(CODEC).

What is claimed is:
 1. A method comprising: displaying, by a firstdevice, virtual reality content; obtaining, by the first device, asynthetic representation of a second device; displaying, by the firstdevice, the synthetic representation of the second device with thevirtual reality content; receiving, by the first device, inputrequesting a change in functionality of the second device; anddisplaying, based on the input received by the first device, a change inthe synthetic representation of the second device, the change in thesynthetic representation of the second device representing the change infunctionality of the second device.
 2. The method of claim 1, whereinthe synthetic representation of the second device includes a syntheticrepresentation of a display of the second device, wherein contentdisplayed by the second device is displayed in the syntheticrepresentation of the display of the second device.
 3. The method ofclaim 1, further comprising: receiving, by the first device, anindication of one or more inputs processed by the second device; anddisplaying, based on the indication of the one or more inputs processedby the second device, an additional change in the syntheticrepresentation of the second device, the additional change in thesynthetic representation of the second device representing an additionalchange in functionality of the second device.
 4. The method of claim 1,wherein the synthetic representation of the second device is overlaidover the virtual reality content.
 5. The method of claim 1, wherein thefirst device includes a virtual reality head mounted display.
 6. Themethod of claim 1, wherein the second device includes a mobile device.7. The method of claim 1, further comprising: obtaining, by the firstdevice, audio content from the second device; and playing, by the firstdevice, the audio content from the second device.
 8. The method of claim1, further comprising: obtaining, by the first device, audio contentcaptured by a microphone of the second device; and playing, by the firstdevice, the audio content.
 9. The method of claim 1, further comprising:obtaining, by the first device, one or more images captured by a cameraof the second device; and displaying, by the first device, the one ormore images with the virtual reality content.
 10. The method of claim 9,wherein the one or more images are part of a video captured by thecamera of the second device.
 11. The method of claim 9, wherein the oneor more images are displayed with the virtual reality content as part ofthe synthetic representation of the second device.
 12. The method ofclaim 9, wherein the one or more images are displayed within a syntheticrepresentation of a display of the second device.
 13. The method ofclaim 9, further comprising: obtaining, by the first device, audiocontent captured by a microphone of the second device; and playing, bythe first device, the audio content while displaying the one or moreimages with the virtual reality content.
 14. The method of claim 1,further comprising: receiving a trigger; and displaying, based onreceiving the trigger, the synthetic representation of the second devicewith the virtual reality content.
 15. The method of claim 14, whereinthe trigger is based on information received from the second device. 16.The method of claim 1, further comprising: receiving a removal trigger;and removing, by the first device based on receiving the trigger, thesynthetic representation of the second device from display.
 17. Anapparatus, comprising: a memory configured to store content for display;one or more processors implemented in circuitry and coupled to thememory, the one or more processors being configured to: cause virtualreality content to be displayed on a display; obtain a syntheticrepresentation of a device; cause the synthetic representation of thedevice to be displayed with the virtual reality content on the display;receive input requesting a change in functionality of the device; andbased on the received input, cause a change in the syntheticrepresentation of the device to be displayed on the display, the changein the synthetic representation of the device representing the change infunctionality of the device.
 18. The apparatus of claim 17, wherein thesynthetic representation of the device includes a syntheticrepresentation of a display of the device, wherein content displayed bythe device is displayed in the synthetic representation of the displayof the device.
 19. The apparatus of claim 17, wherein the one or moreprocessors are configured to: receive an indication of one or moreinputs processed by the device; and based on the indication of the oneor more inputs processed by the device, cause an additional change inthe synthetic representation of the device to be displayed on thedisplay, the additional change in the synthetic representation of thedevice representing an additional change in functionality of the device.20. The apparatus of claim 17, wherein the synthetic representation ofthe device is overlaid over the virtual reality content.
 21. Theapparatus of claim 17, wherein the apparatus includes a virtual realityhead mounted display, the virtual reality head mounted display includingthe display.
 22. The apparatus of claim 17, wherein the device includesa mobile device.
 23. The apparatus of claim 17, wherein the one or moreprocessors are configured to: obtain audio content from the device; andplay the audio content from the device.
 24. The apparatus of claim 17,wherein the one or more processors are configured to: obtain audiocontent captured by a microphone of the device; and play the audiocontent.
 25. The apparatus of claim 17, wherein the one or moreprocessors are configured to: obtain one or more images captured by acamera of the device; and cause the one or more images to be displayedwith the virtual reality content.
 26. The apparatus of claim 25, whereinthe one or more images are part of a video captured by the camera of thedevice.
 27. The apparatus of claim 25, wherein the one or more imagesare displayed with the virtual reality content as part of the syntheticrepresentation of the device.
 28. The apparatus of claim 25, wherein theone or more images are displayed within a synthetic representation of adisplay of the device.
 29. The apparatus of claim 25, wherein the one ormore processors are configured to: obtain audio content captured by amicrophone of the device; and play the audio content while the one ormore images are displayed with the virtual reality content.
 30. Theapparatus of claim 17, wherein the one or more processors are configuredto: receive a trigger; and based on receiving the trigger, cause thesynthetic representation of the device to be displayed with the virtualreality content.
 31. The apparatus of claim 30, wherein the trigger isbased on information received from the device.
 32. The apparatus ofclaim 17, wherein the one or more processors are configured to: receivea removal trigger; and based on receiving the trigger, cause thesynthetic representation of the device to be removed from display by thedisplay.
 33. A computer-readable storage medium storing instructionsthat when executed cause one or more processors of a first device to:cause virtual reality content to be displayed by the first device;obtaining, by the first device, a synthetic representation of a seconddevice; cause the synthetic representation of the second device to bedisplayed with the virtual reality content; receiving, by the firstdevice, input requesting a change in functionality of the second device;and based on the input received by the first device, cause a change inthe synthetic representation of the second device to be displayed by thefirst device, the change in the synthetic representation of the seconddevice representing the change in functionality of the second device.